A vacationer snaps a photograph of her friend lounging by the pool at their resort hotel. When the friends are looking through their photographs a few days later, they notice that a bystander strayed into the background of the scene when the picture was taken, thus spoiling the intended shot. They wish to remove the bystander from the photograph. The removal of an unwanted object appearing in a digital image is one example of an image manipulation operation that can be performed by a wide range of existing image editing software applications. Ideally, the unwanted object is replaced with a visually plausible background to produce a modified image that seems reasonable to the human eye. The particular algorithm used to remove the unwanted object is selected based on the size of the object to be removed from the source image and the appearance of the background scene. For example, texture synthesis algorithms, which seek to replicate a small sample texture source, work well when applied to fill large regions. Image “inpainting” algorithms, which fill holes in images by propagating linear structures into the target region via diffusion, work well in image restoration applications, such as to remove speckles, scratches, or overlaid text. Examples of existing image editing software applications that use algorithms such as these for object removal include Adobe Photoshop (Adobe Systems Incorporated, San Jose, Calif.), Corel Paint Shop Pro (Corel Corporation, Ottawa, Canada), and Autodesk SketchBook (Autodesk, Inc., San Rafael, Calif.).